Decoding Frames

ABSTRACT

A system for decoding a data stream, comprising: a first decoder configured to decode the data stream at a first rate so as to generate a first stream of frames for playback and arranged to continue generating the first stream despite encountering an error in a particular frame; a second decoder operable to decode the data stream at a second rate so as to generate a second stream of frames; and a controller configured to: detect the error and cause the second decoder to decode the data stream from the particular frame in dependence on error correction data, the second rate being faster than the first rate such that the second stream catches up with the first stream; determine when the second decoder catches up with the first decoder; and cause the second decoder to operate at the first rate so as to generate the second stream for playback.

BACKGROUND OF THE INVENTION

This invention relates to a data processing system arranged to perform aresynchronisation mechanism for a decoder.

Real-time streaming of multimedia content over the internet has becomean increasingly common application in recent years. A wide range ofmultimedia applications, such as on-demand TV, live TV viewing, videoconferencing, net meetings, video telephony and many others rely onend-to-end streaming solutions. Unlike a “downloaded” video file, whichmay be retrieved first in “non-real” time and viewed or played backlater, streaming video applications require a video source to encode andto transmit a video signal over a network to a video receiver, whichmust decode and display the video signal in real time. The receivingdevice receives encoded video data packets from the network andtransfers the packets to a video decoder for decoding.

Compression techniques for transmitting video data can use so-calledreference frames. When compressing blocks of video data, the encodingprocess can generate intra frames (I-frames). An I-frame is a compressedversion of a frame which can be decompressed using only the informationin the I-frame itself, and without reference to other frames. They aresometimes referred to as key frames. Another type of frame can also begenerated, which are sometimes referred to as inter or predictive frame(P-frames), which are generated by predictive inter frame coding based,directly or indirectly, on a reference frame. The reference frame can bethe preceding frame, or it could be a different earlier or later framein a sequence of frames.

During the process of I-frame and P-frame coding, information in theimage is processed block-wise and a Discrete Cosine Transform (DCT) isapplied to each block. The resulting DCT coefficients consist ofcoefficients corresponding to the strength of various frequencies in theblock. The coefficients of each block are quantized with various levels,so as to achieve the desired trade-off of quality and bit-rate. Theencoder then reconstructs the quantized frame by applying inversequantization and inverse DCT in the exact same way as a decoder would,and uses it as a potential reference frame for subsequent frames. Thereplication of the decoder functionality helps keep an encoder anddecoder synchronized.

Problems can arise when a streaming video signal is transmitted acrossnetworks, such as the Internet. For example, significant packet lossrate across the transmission network often requires re-transmission ofthe lost packets. Typically, a lost data packet needs to be recoveredprior to the time the corresponding frame must be decoded. If the lostpacket is not received, the current frame being processed as well as thesubsequent frames can be adversely affected because of the predictivecoding.

Loss of packets can cause a loss in the synchronisation of the states ofthe encoder and decoder. This can cause artifacts in the displayedframes due to corrupted reference frame(s) arising due to the missingpackets of the encoded frames. Due to real-time play constraints in someapplications such as video telephony, the decoder may have to decode anincomplete frame and thus it will lose state synchronization with theencoder. After the incomplete frame has been decoded, any packetsbelonging to the decoded frame that arrive subsequent to decoding of theframe, e.g. due to late arrival or FEC repair or retransmitted packets,cannot help regain decoder state synchronization.

Typically, re-synchronization of the decoder state can be achieved byrequesting an Instantaneous Decoder Refresh (IDR), enabling a ReferencePicture Selection (RPS) feature or waiting for a key frame.

An IDR can be requested by a receiver, which when received without lossand subsequently decoded, completely synchronizes the encoder-decoderstate. However, frequent IDR requests can lead to bandwidth beingexhausted leading to poor quality, frame freezes and violations ofconstant bitrate constraints for some applications such as videotelephony.

The RPS feature allows a receiver, upon encountering an incompleteframe, to indicate to an encoder that one of its previously receivedcorrect frames is to be used as a new reference frame for the next framein encoder pipeline. However, RPS suffers from issues when the roundtrip delay between the encoder and decoder is greater than the durationfor which reference frames are stored in encoder. RPS also tends toreduce encoder efficiency due to prediction from older frames.

Key frames, which can be protected by forward error correction (FEC),ensure regular encoder-decoder synchronization, provided they do nothave missing packets. Packet losses in non-key frames will temporarilycause loss of synchronization until the next key frame. However, keyframes tend to reduce encoding efficiency due to prediction from olderframes. Furthermore, the FEC feature is proprietary and is not supportedin standards.

Other approaches used to solve the decoder state synchronization probleminclude halting decoding until frame completion. This mechanism forincomplete frame handling involves waiting for retransmitted/repairpackets to arrive before continuing to decode the incomplete frame.However, this causes the displayed video to be paused and appearstuttered. Another approach would be to increase the buffering delay.This estimates the typical missing packet retransmission time andincreases the buffering delay. However, this approach leads to loss inreal-timeliness and interactivity, deeming it unsuitable forapplications such as video telephony and video conferencing.

Furthermore, in Multipoint Control Unit (MCU) centric videoconferencing, where multiple receivers can experience packet loss atdifferent times, frequent IDR frames may be generated, leading to poorquality. Decoder synchronization by RPS selection also may not befeasible with a single encoder instance, unless multiple encoderinstances, specific to each end-point are used. In MCU-less videoconferencing, where one sender sends the same packets to multiplereceivers, it may be sub-optimal to generate IDR frames for eachreceiver's packet loss. In live video User Datagram Protocol (UDP)streaming to multiple receivers, similar real-time constraints andissues also apply.

SUMMARY OF THE INVENTION

According to a first aspect of the present invention there is provided adata processing system for decoding a data stream, the data processingsystem comprising: a first decoder configured to decode the data streaminto a sequence of media frames at a first decode rate so as to generatea first media stream for playback in real-time, the first decoder beingarranged to continue generating the first media stream for playbackdespite encountering an error in decoding a particular frame of thesequence; a second decoder operable to decode the data stream into asequence of media frames at a second decode rate so as to generate asecond media stream; and a controller configured to detect the errorand, responsive to that detection, cause the second decoder to decodethe data stream from the particular frame in the sequence of mediaframes, the second decoder being arranged to decode from the particularframe in dependence on error correction data received over the datastream for the particular frame, wherein the second decode rate isfaster than the first decode rate such that the sequence of frames ofthe second media stream catches up with the sequence of frames of thefirst media stream and the controller is further configured to:determine when the sequence of frames from the second decoder catches upwith the sequence of frames from the first decoder; and responsive tothat determination, cause the second decoder to operate at the firstdecode rate so as to generate the second media stream for playback inreal-time.

The controller may be further configured to cause the first decoder tocease decoding the media data stream responsive to its determination asto when the sequence of frames from the second decoder catches up withthe sequence of frames from the first decoder.

Preferably, the controller is configured to determine when the sequenceof frames from the second decoder catches up with the sequence of framesfrom the first decoder by estimating from the first and second decoderates when the first and second decoders will decode the same frame. Thesecond decode rate may be between two and four times greater than thefirst rate. Preferably, the second decode rate is about two timesgreater than the first rate.

The controller may be configured to determine when the sequence offrames from the second decoder catches up with the sequence of framesfrom the first decoder by determining a time difference between a timewhen the particular frame is decoded at the first decoder and a timewhen the particular frame is decoded at the second decoder, and dividingthe time difference by a relative rate subtracted by 1, the relativerate being the second decode rate divided by the first decode rate.

Preferably, the controller is configured to determine the second decoderate from the processing resources available at the data processingsystem for the second decoder.

Preferably the controller is configured to, responsive to the errordetection, initialise the second decoder using context data from thefirst decoder.

The context data may include one or more of the last complete frames oneor more of the last complete frames of the first media stream, stateinformation of the first decoder from the up to the point at which thelast complete frame was decoded, the first frame as generated by thefirst decoder, and state information of the first decoder from the pointat which the error was encountered.

The error encountered by the first decoder may be data missing from thedata stream, or corrupted data of the data stream, or an encoder error.

Preferably, the first decoder is configured to decode said particularframe so as to generate a frame with artifacts for playback.

Preferably, the first decoder is configured to, on continuing togenerate the first media stream for playback after encountering theerror decoding the particular frame, decode frames subsequent to theparticular frame so as to generate frames for playback which potentiallyinclude artifacts.

The data processing system may further comprise a communications portfor receiving the data stream.

According to a second aspect of the invention there is provided machinereadable code for generating the data processing system.

According to a third aspect of the invention there is provided a machinereadable storage medium having encoded thereon non-transitory machinereadable code for generating the data processing system.

According to a fourth aspect of the invention there is provided a methodof decoding a data stream, the method comprising: at a first decoder,decoding the data stream into a sequence of media frames at a firstdecode rate so as to generate a first media stream for playback inreal-time, and continuing to generate the first media stream forplayback despite encountering an error in decoding a particular frame ofthe sequence; detecting the error and, responsive to that detection,causing a second decoder to decode the data stream from the particularframe in the sequence of media frames, at the second decoder, decodingthe data stream into a sequence of media frames at a second decode rateso as to generate a second media stream, the second decoder decodingfrom the particular frame in dependence on error correction datareceived over the data stream for the particular frame, wherein thesecond decode rate is faster than the first decode rate such that thesequence of frames of the second media stream catches up with thesequence of frames of the first media stream; determining when thesequence of frames from the second decoder catches up with the sequenceof frames from the first decoder; and responsive to that determination,causing the second decoder to operate at the first decode rate so as togenerate the second media stream for playback in real-time.

According to a fifth aspect of the invention there is provided acomputer readable code adapted to perform the steps of the method whenthe code is run on a computer.

According to a sixth aspect of the invention there is provided acomputer readable storage medium having encoded thereon the computerreadable code.

DESCRIPTION OF THE DRAWINGS

The present invention will now be described by way of example withreference to the accompanying drawings, in which:

FIG. 1 is a schematic diagram of a device capable of receiving anddecoding a sequence of video frames;

FIG. 2 illustrates an example of decoding a sequence of frames havingone instance of error at the receiving device;

FIG. 3 illustrates another example of decoding a sequence of frameshaving two instances of error at the receiving device;

FIG. 4 is a flowchart of a method of decoding a sequence of frames; and

FIG. 5 is a flowchart illustrating the processing of packets.

DETAILED DESCRIPTION OF THE DRAWINGS

The following description is presented to enable any person skilled inthe art to make and use the invention, and is provided in the context ofa particular application. Various modifications to the disclosedembodiments will be readily apparent to those skilled in the art.

The general principles defined herein may be applied to otherembodiments and applications without departing from the spirit and scopeof the present invention. Thus, the present invention is not intended tobe limited to the embodiments shown, but is to be accorded the widestscope consistent with the principles and features disclosed herein.

Embodiments of the present invention provide an improved mechanism forresynchronising a decoder with an encoder. In the examples below, amechanism for resynchronising a video decoder with a correspondingencoder is described. However, the mechanism can be applied to othertypes of media decoders that employ inter-frame predictive codingtechniques that enable the playback of the media from a data stream.

FIG. 1 depicts a receiving device 10, which may be any suitable devicethat is capable of consuming packet based data such as a computer,smartphone, videophone, etc. The receiving device 10 comprises acommunications port 11 for connection to a communications network 17such as the internet or other packet based networks. The receivingdevice 10 can transmit and/or receive data packets to and/from thecommunications network 17 via the communications port 11.

A transmitting device (not shown) can encode video data for transmittingover the communications network 17 to the receiving device 10. The videodata may be encoded according to a video coding standard such as H.261,H.262 (MPEG-2 or ISO/IEC 13818-2), H.263 or H.264 (AVC or ISO/IEC14496-10) standards or the MPEG-1 (ISO/IEC 11172-2), MPEG-4 Visual(ISO/IEC 14496-2) or SMPTE 421M standards. The encoded video data cantake the form of a bit stream comprising a series of frames which aretransmitted in the form of packets. The frames can include P-frames andI-frames. P-frames contain data representing the difference between theframe and one or more reference frames. I-frames are frames representingthe difference between pixels within a frame, and as such can be decodedwithout reference to another frame.

The receiving device 10 comprises a first decoder 12 which can decodethe encoded video data received at the communications port 11. FIG. 2illustrates the decoding of a sequence of frames received at thereceiving device 10. The receiving device 10 receives a stream of one ormore packets carrying data encoding frame 101, which may be a P- orI-frame. Frame 101 may be considered to be a “complete” or “lossless”frame if all (or more than a threshold level) of the packets required todecode the frame are received by the receiving device 10. The state ofthe first decoder 12 is thus said to be in synchronisation with theencoder because the frame has been successfully decoded by the firstdecoder 12 and is at least substantially the same as the frame encodedby the encoder. The first decoder 12 outputs decoded frame 101, whichcan then be displayed. The receiving device 10 may comprise a display ordisplay input 16 for displaying the decoded frames.

The first decoder 12 then decodes the next frame 102 in the sequence offrames. The first decoder 12 decodes the sequence of frames at a firstrate, which may be the real-time rate at which the video frames are tobe displayed (at least when the encoder reaches its steady state). Thefirst rate may be predetermined or set by, for example, the encoder atthe transmitting device. In FIGS. 2 and 3, arrow 124 indicates thedirection of the flow of time and the spacing between the framesindicates the rate at which the frames are decoded. In the example shownin FIG. 2, frame 102 is successfully decoded and the frame is output fordisplay.

If the decoder 12 does not receive all of the data defining a frame intime for when the decoder 12 is due to decode that frame—for example,because one or more data packets carrying the frame data go missing orbecome corrupted—then the decoded frame generated by the decoder 12 willbe incomplete and include errors. Such errors would typically beapparent in the decoded frame as one or more artefacts and/or missingportions of the frame. In the example shown in FIG. 2, frame 103 in thesequence of frames is erroneous and exhibits a partial loss 113 in theframe because not all of the data required to completely decode theframe was made available to the decoder 12 by the time frame 103 was dueto be decoded.

As mentioned above, in some applications such as video conferencing, itmay be preferable to maintain a real-time display rather than pause thedisplay while data to correct the error or a new key frame is awaited.The decoder 12 is configured to decode as much of incomplete frame 103as possible so that it can be displayed at the appropriate time withoutany delay. Thus the real-time rate of displaying (and potentiallydecoding) the sequence of frames is maintained. Frame 103 could be an Iframe, or a P frame based on a preceding frame, e.g. frame 102.

Due to the incomplete frame 103, the first decoder 12 can be said tohave lost synchronisation with the encoder as the decoded frame isdifferent (due to the partial loss 113) to the frame that was encoded.This loss in synchronisation can cause errors in subsequent frames. Inthe example shown in FIG. 2, frame 104 is a P frame that is based onpreceding frame 103. However, frame 103 has a partial loss 113 whichwill propagate into frame 104 because frame 104 is a P frame based onframe 103. The partial loss 114 would continue to propagate throughsubsequent frames 105-107 (as shown in the figure) until a new “key”frame is received whose ability to be decoded does not depend on priorframes. Thus frames 104 to 107 will also be decoded and displayed withpartial losses 114 to 117.

The missing or corrupted data that leads to the error 113 in frame 103may become available at a later time due to, for example, subsequentretransmission of the missing data itself or other data sufficient topermit recovery of the missing data for frame 103. However, at thislater time, frame 103 would be likely to be out-of-date and unsuitablefor decoding and display in a real-time video stream. In the exampleshown in FIG. 2, the data required to complete frame 103 arrives at thereceiving device 10 at about the time that frame 105 is decoded by thedecoder 12, and is too late for the frame 103 to be decoded by thedecoder 12 and displayed without error. This delay between frame 103being partially decoded for display in the real-time video stream andthe time at which the missing data is received is indicated by arrow 125in FIGS. 2 and 3.

Receiving device 10 further comprises a second decoder 13 and acontroller 14 which can help recover synchronisation with the encoder.The controller 14 may be a separate entity to, and coupled to, thedecoders 12 and 13. Alternatively, one or both decoders 12 and 13 may beconfigured to carry out some or all of the functionalities of thecontroller described herein.

The controller 14 is arranged to detect errors. The controller 14 candetect errors such as missing or corrupted data for a frame, there-transmitted data (e.g. error correction data or late arriving datapackets carrying missing frame data) and a decoder 12 or 13 flagging upan error while decoding. On arrival of error correction data, such asthe missing data, the controller 14 initialises second decoder 13 andcauses the decoder 13 to complete the decode of frame 103 so as to formcomplete frame 103 a with error 113 being corrected 113 a. Frame 103 ais not displayed because the incomplete version of it decoded at thefirst decoder 12 has already been displayed. Decoder 13 then continuesto decode incoming data so as to generate a stream of complete frames104 a-109 a which do not include the errors, caused by error 113,propagating through the stream of frames generated by decoder 12.However, the frames for display are for the moment provided by thestream of incomplete frames from decoder 12, which runs concurrentlywith decoder 13.

Each decoder 12 and 13 maintains context data necessary for the decodingof an encoded video stream into a sequence of decoded frames. Whendecoding a video stream that makes use of intermediate and referenceframes, between reference frames such context data would typicallyinclude the most recently decoded frame with respect to which thechanges expressed in a subsequent intermediate frame are applied. Thecontext data is saved into a separate region of memory in response toevents such as errors in the decoder, or missing data for a frame.

On receiving the data missing from frame 103, the second decoder 13 isinitialised with a saved context of decoder 12 that includes theinformation necessary for decoder 13 to decode frame 103 a and continuethe decode processing of the stream of frames from frame 103 a. In thecase frame 103/103 a is a P frame, the context may for example includethe previous complete frame, 102.

The context of decoder 12 could be replicated and saved in response aloss in the encoder-decoder synchronisation. The loss in encoder-decodersynchronisation may be caused by, for example, the decoder 12 beingunable to perform a complete decode of a frame, due to, for example,missing one or more or all data packets for that frame, missing anI-frame that a P-frame to be decoded is dependent on, or any otherreason that would cause a loss in synchronisation. Either the decoder 12or controller 14 might be configured to cause the context data to bereplicated and stored, suitably at a memory 15. The saved context datamight be the context data associated with the last complete frame thatwas successfully decoded (e.g. frame 102 in FIG. 2). The saved contextdata could include the state of the decoder 12 at the time frame 102 wasdecoded, the received encoded data for frame 102, the decoded frame 102and other suitable data that would allow continuation of the decodingprocess from frame 102 at an arbitrary later time. Alternatively, thesaved context data might be context data associated with the frame whichthe decoder 12 was unable to fully decode (e.g. frame 103) such that thesecond decoder 13 can utilise some of the processing performed bydecoder 12 in re-performing the decode of frame 103.

Context data could be maintained for the last complete frame that wassuccessfully decoded by arranging that the context data associated witheach complete frame that is successfully decoded is automatically saved.Again, either the decoder 12 or controller 14 might be configured tocause the context data to be stored, suitably at the memory 15.

It can be advantageous to arrange that the controller 14 itselfdetermines that the decoder 12 is unable to perform a complete decode ofa frame. This could be implemented by arranging the controller 14 todetect missing data in the video packet stream from the transmittingdevice as a result of, for example, packet loss during transmission overthe network 17, late arrival of data packets, receiving corrupted datapackets, or by any other suitable means. On detecting an error, thecontroller 14 is preferably configured to send a request to thetransmitting device to retransmit the missing data—e.g. by requestingretransmission of data packets carrying the missing data.

In order to enable decoder 13 to catch up with decoder 12, seconddecoder 13 is configured to run at a faster rate than decoder 12, whichneed only maintain a decode rate sufficient to provide frames fordisplay at real-time. In this way, the sequence of frames 103 a to 107 agenerated by the second decoder 13 catches up with (i.e. converges on)the sequence of frames 103 to 107 generated by the first decoder 12. Atsome point, both decoders will be decoding the same frame. In theexample shown in FIG. 2, this occurs when the generating of frame 107 bydecoder 12 coincides with the generating of frame 107 a by decoder 13.At this point, the rate that the frames are decoded at the seconddecoder 13 is decreased to match the rate at which the first decoder 12is being operated.

Once decoder 13 has caught up with decoder 12, the stream of frames fordisplay is switched from displaying the stream of frames from decoder 12to displaying the stream of frames 108 a, 109 a from decoder 13. Thus,switching of the streams for display from the first decoder 12 to thesecond decoder 13 preferably occurs such that the next frame due to bedisplayed in the sequence of frames after the streams coincide is aframe from the second decoder 13. Preferably, the switching of thestreams for display occurs such that the rate at which the frames aredisplayed (e.g. the real-time rate) can be maintained. For example, ifdecoder 13 starts decoding frame 107 a at the same time as decoder 12starts decoding frame 107, then frame 107 a will have been decoded intime for it to be displayed and so the stream for display can beswitched so that frame 107 a is displayed instead of frame 107. On theother hand, if decoder 13 starts decoding frame 107 a at a time afterdecoder 12 starts decoding, then it is likely that frame 107 a may notbe decoded in time for it to be displayed and so frame 107 is displayedand then the stream for display is switched to frame 108 a onwards.

After the switch of the stream for display, decoder 12 is configured tostop decoding frames and may be deactivated or put into a standby mode,which allows the receiving device 10 to save on resources. At the pointwhen the stream of frames from the second decoder 13 is displayed,encoder-decoder state synchronization is said to be achieved.

During the catch-up phase 126 the first and second decoders 12 and 13are decoding frames at the same time, but at different rates. During thecatch up phase 126, the erroneous stream of frames (103-107) decoded atthe first decoder 12 is displayed and the speeded up stream of frames(103 a-107 a) decoded at the second decoder 13 is not displayed. Oncethe second decoder 13 has caught up with the first decoder 12, thecontroller 14 switches the stream of frames to be displayed from thestream of frames decoded at the first decoder 12 to the stream of framesdecoded at the second decoder 13.

The faster rate of the second decoder 13 during the catch-up phase 126can be varied depending upon available processing resources of thereceiving device 10. The controller 14 can determine the processingresources available and set the decoding rate of the second decoder 13accordingly. The decoding rate at the second decoder 13 may beproportional to the amount of processing resources available. Increasingthe decoding rate of the second decoder 13 would shorten the catch upperiod 126. However, the amount that this rate is increased by may belimited by the amount of processing resources available. Thus there is atrade-off between additional CPU usage and the encoder-decoderresynchronization time.

The controller 14 can determine the total amount of processing resourcesavailable for decoding frames. The controller 14 can then determine theminimum amount of processing resources required to decode frames atdecoder 12 at the real-time rate and subtract this minimum amount fromthe total amount to determine an amount of processing resourcesavailable for decoder 13. The faster decode rate can then be determinedfrom the amount of processing resources available for decoder 13. Thecontroller 14 can then set decoder 13 to decode at the determined rate.The faster decode rate of decoder 13 can be determined in dependence on,for example, available CPU and memory resources, the size of the frames,the frame rate at which the frames are to be displayed, etc. Asdescribed below, the catch-up time 126 is dependent on the relativedecoding rates of the first and second decoders 12 and 13. Thus thecatch-up time 126 can also be determined in dependence on the availableresources.

Preferably, the faster rate at the second decoder 13 during the catch upperiod is between about two to about four times the “real-time” rate ofthe first decoder 12. The controller 14 can estimate the expectedcatch-up time 126 using the difference in the decoding rates at thedecoders 12 and 13. Typically, a setting of two times the first decoderrate will deterministically bind the maximum decoder CPU usage to threetimes a single decoder CPU usage for the duration of the catch up phase126.

Based on the frame decoding rates of the first and second decoders 12and 13 and the amount of time between decoding frame 103 at the firstdecoder 12 and decoding frame 103 a at the second decoder 13, thecontroller 14 can determine an approximate time when the streams ofdecoded frames from the first and second decoders 12 and 13 willcoincide.

The catch up time 126 can be calculated using the following equation:

$\begin{matrix}{{{Catchup}\mspace{14mu} {time}} = {\left( {{TS}_{current} - {TS}_{recovered}} \right) \times \left( {\frac{1}{rate} + \frac{1}{{rate}^{2}} + \frac{1}{{rate}^{3}} + \ldots}\mspace{14mu} \right)}} & (1)\end{matrix}$

Where TS_(current) is the timestamp (in, e.g., micro-seconds) of thecurrent frame decoded at the decoder 12 at the start of the catch upperiod (frame 105) or a time when the second decoder 13 decodes frame103 a, TS_(recovered) is the timestamp (in, e.g., micro-seconds) of theold frame that was erroneous (frame 103) and the “rate” is the rate ofthe second decoder 13 relative to the first decoder, which can becalculated by dividing the decoding rate at the second decoder 13 by thedecoding rate at the first decoder 12.

Equation 1 can be explained using the following example. Assume thedifference in TS_(current) and TS_(recovered) is 1000 ms and the “rate”of the second decoder 13 is 2 times the speed of the first decoder 12.The amount of frames decoded at the first decoder 12 in 1000 ms will bedecoded in 1000×½ (first term in equation)=500 ms at the second decoder13. However, by this time the first decoder 12 would have advanced by500 ms. So now the amount of frames decoded at the first decoder 12 in500 ms will take the second decoder 13 500*½=250 ms=1000*¼ (second termin equation) to decode the same amount. Again 250 ms at 2× speed willtake 125 ms=1000*⅛ (third term in equation), and 125 ms at 2× speed willtake 62.5 ms=1000* 1/16 (fourth term in equation) and so on. Thisrepresents a geometric progression.

Equation 1 in the form of a geometric progression is:

$\begin{matrix}{{{a + {ar} + {ar}^{2} + {ar}^{3} + {ar}^{4} + \ldots} = {{\sum\limits_{k = 0}^{\infty}\; {ar}^{k}} = \frac{a}{1 - r}}},{{{for}\mspace{14mu} {r}} < 1}} & (2)\end{matrix}$

Alternately,

$\begin{matrix}{{{{{ar} + {ar}^{2} + {ar}^{3} + {ar}^{4} + \ldots} = {{{\sum\limits_{k = 0}^{\infty}\; {ar}^{k}} - a} = {{\frac{a}{1 - r} - a} = \frac{ar}{1 - r}}}},{{{for}\mspace{14mu} {r}} < 1}}{{{{where}\mspace{14mu} a} = \left( {{TS}_{current} - {TS}_{recovered}} \right)},{{{and}\mspace{14mu} r} = \frac{1}{rate}}}} & (3)\end{matrix}$

Substituting provides the following equation for determining the catchup time:

$\begin{matrix}{{{Catchup}\mspace{14mu} {time}} = \frac{\left( {{TS}_{current} - {TS}_{recovered}} \right)}{{rate} - 1}} & (4)\end{matrix}$

The calculated catch up time is the amount of time required to thesecond decoder 13 to catch up with the first decoder 12. Thus the pointin time when catch up is complete (hereinafter referred to as the “catchup completion time”) is the time when the second decoder 13 beingsdecoding frame 103 a in addition to the calculated catch up time. Usingthe calculated catch up completion time, the controller 14 can determinewhen to switch the stream of frames for display from the first decoder12 to the second decoder 13. For example, the controller 14 cancalculate the catch up completion time and switch the stream of framesfrom the first decoder 12 to the second decoder 13 so that the switchedsteam begins at the first frame decoded at the second decoder 13 afterthe catch up completion time.

During the catch-up phase 126, if the first decoder 12 receives datasufficient to correct loss of synchronization (e.g. an InstantaneousDecoder Refresh, IDR), the decoder-encoder synchronization isautomatically regained and so the catch-up phase 126 is terminated andthe second decoder 13 is deactivated.

In some scenarios, processor resources may be insufficient for catch-upwithin a reasonable time due to the effective decoding rate of thesecond decoder 13 being less than or approximately the same as that ofthe first decoder 12. Under such conditions, the controller 14 can beconfigured to terminate the second decoder 13 and, if it hasn't alreadydone so, request resynchronisation of the decoder (e.g. request an IDR).

FIG. 3 illustrates a scenario where the second decoder 13 encounters aframe 105 a comprising a second error 118 during the first catch upphase 126. As discussed above, context data for the second decoder 13and associated with frame 104 a is saved. This allows a third stream offrames (105 b-112 b) initialised using context data saved at frame 104 ato begin at the first decoder 12. When the first catch up phase 126 hascompleted, the stream of frames displayed is switched by the controller14 from the first stream (101-107) outputted from the first decoder 12to the second stream outputted from the second decoder 13 from frame 108a.

Because of the second error 118 introduced at frame 105 a, displayedframes 108 a to 110 a outputted by the second decoder 13 are incompleteand comprise second errors 121 to 123 respectively which have propagatedthrough the stream from the respective errors in frames 105 a to 107 a.When the first catch up phase 126 is completed and decoder 13 isproviding frames for display, the first decoder 12 can be used togenerate a new stream of frames 105 b-112 b that are free from theerrors introduced by error 118.

The second catch up phase 127 can begin at decoder 12 once the missingdata causing error 118 has been received so that a complete frame 105 bcan be decoded with error 118 being corrected 118 b. In an analogousfashion to the first catch-up phase 126, in the second catch-up phase127 the first decoder 12 is operated at a faster rate than seconddecoder 13. The second catch up phase 127 is initialised using thesecond decoder context data saved at frame 104 a. When the second catchup phase 127 ends at frame 110 a/b, the decoding rate at the firstdecoder 12 is decreased to match that of decoder 13 and the controller14 switches the frames to be displayed from the stream of frames 108 aonwards generated at decoder 13 to the stream of frames 111 b onwardsgenerated at decoder 12. After the switch of the stream for display,decoder 13 can then cease decoding of any frames. At the end of thesecond catch-up phase 127, decoder 12 can be said to once again be insynchronization with the encoder.

Each of the first and second decoders 12 and 13 may comprise its owninput and/or output buffers (not shown). Data for decoding into framesis directed to the active decoders by the controller 14. Typically onlyone decoder will be active until a frame cannot be completely decodedand a catch-up phase is entered where both decoders are active. Forexample, with reference to FIG. 2, the controller 14 causes frame datareceived at the communication port 11 relating to frames 101 to 107 tobe sent to the input buffer of the first decoder 12 for decoding, butdata relating to frames 103 to 107 to be sent to both decoders (12 and13) during the catch-up phase 126.

The first and second decoders described herein may be separate decoderunits, different decoder pipelines of a single decoder unit, or a singledecoder unit arranged to concurrently support the first and seconddecoders. Each of the first and second decoders can have their owninput/output buffers, as described in the paragraph above. In the casethat the first and second decoders are supported at a single decoderunit, the separate decode streams of the first and second decoders couldbe concurrently supported in any suitable manner. For example, such adecoder unit could be configured to alternate between running the firstdecoder and running the second decoder at a common decoder pipeline,perhaps by alternately allocating slices of processing time to each ofthe first and second decoders. Thus, in the case that the first andsecond decoders are supported at a single decoder unit: (i) a firstinput buffer, the decoder unit and a first output buffer could operateas the first decoder; and (ii) a second input buffer, the same decoderunit and a second output buffer could operate as the second decoder 13.The behaviour of first and second decoders supported at a single decoderunit can be controlled through the use of appropriate buffer synchronouscommands.

In the examples described herein, two decoders are provided. However, itwill be appreciated that the decoding and synchronising method describedherein can be readily extended to receiver devices having more than twodecoders.

An exemplary method for decoding a video data stream 401 is outlined inthe flowchart of FIG. 4.

At step 402, the data stream is decoded into a sequence of frames at thefirst decoder 12 at a real-time rate. The first decoder 12 generates afirst stream of decoded frames, which are to be displayed.

At step 403, an error is detected by the first decoder 12. The errorcould be, for example, missing or corrupted data that will cause a frameto be partially or wholly incomplete.

At step 404, the first decoder 12 continues to decode the erroneousframe and the frames subsequent to the erroneous frame so that they canbe displayed. By allowing the first decoder 12 to continue decodingerroneous frames, the video can continue to be displayed without beinghalted or frozen while the error is being corrected.

At step 405, the missing data that allows the complete frame to berecovered is received.

At step 406, the second decoder 13 decodes the frame using the missingdata and frames subsequent to that frame are also decoded at the seconddecoder 13 to generate a second stream of decoded frames. The frames aredecoded at the second decoder 13 at a rate that is greater than thedecode rate of the first decoder 12 such that the stream of frames atthe second decoder 13 converges with the stream of frames being decodedat the first decoder 12.

At step 407, it is determined when the first and second streams ofdecoded frames become coincident. The streams become coincident when thefirst and second decoders decode the same frame in the sequence offrames at approximately the same time.

At step 408, based on the determination, the decoding rate of the seconddecoder 13 is reduced to the real-time rate from the frame at which thefirst and second streams coincide. At step 409, the stream of decodedframes for display is switched from the first stream to the secondstream from the second decoder 13. The displayed frames from the secondstream do not contain the error as the sequence of frames decoded at thesecond decoder 13 began with the corrected frame. The state of thedecoder used at the receiving device 10 has thus been resynchronisedwith the encoder without halting or pausing the display.

FIG. 5 depicts an exemplary method for processing packets 501 receivedas part of the data stream received at the receiving device 10.

At step 502, the frame that the packets belongs to is identified. Thepackets may have a timestamp which identifies the frame that the packetsare for. If the packet is a recovered or retransmitted packet, then theprocess moves on to step 512. Otherwise the process moves on to step503.

At step 503, the packets are stored and arranged according to the framethat the packet belongs to. The packets for the same frame can be storedand arranged in a corresponding frame container.

At step 504, it is determined whether the first decoder 12 has beeninitialised and started to decode frames. If the decoder 12 has not beeninitialised then the process moves on to step 505. If the decoder hasbeen initialised and is therefore ready to decode frames, then theprocess moves on to step 507.

At step 505, if a frame container for a frame comprises the informationnecessary to initialise the decoder in order to decode a sequence offrames (e.g. Sequence Parameter Set (SPS), Picture Parameter Set (PPS)and IDR) then the decoder 12 is initialised using that information atstep 506. If the frame container does not comprise that information,then the process loops back until that information is received. When thedecoder 12 has initialised, the next frame container is checked at step507. If that next frame is not an IDR frame or I-frame, the processmoves on to step 508.

At step 508, if all of the packets for the frame have been received tocomplete the frame then the process moves on to step 509 where the frameis decoded at the real-time rate and is outputted from the decoder 12 sothat it can be displayed. The process then loops back to step 507 forthe next frame in the sequence of frames.

If, at step 508, not all of the packets for a complete frame have beenreceived, then the process moves on to step 510. At step 510, it isdetermined if there are resources available for another decoder contextto be saved. In this example, an active first or second decoder 12 or 13can each save only one decoder context. For example, referring to FIG.3, the first decoder 12 initially operates with a first contextassociated with frame 101. Then, a second context associated with frame102 is saved due to the error 113 in frame 103. The second context isused to initialise the second decoder 13. The first decoder 12 does notsave another context in response to second error 118 in frame 105 as thefirst decoder has already saved the second context which is being usedby the second decoder 13. Instead, the second decoder 13 saves a thirdcontext associated with frame 104 a in response to the second error 118in frame 105 a. If another error was encountered after saving the thirdcontext (e.g. another error in frame 106 a) then another context cannotbe saved as the second decoder 13 has already saved the third contextand so an IDR is requested.

Preferably, the number of decoder contexts that can be saved andutilised at a point in time is limited to N+1 contexts for N decoders soas to save on resources. If there are adequate resources, then thecontext of the first decoder 12 is saved at step 511. Preferably, whenmore than one decoder is active, only the last decoder in the catch uphierarchy is allowed to save a context, if it has not already done so.All other decoders will continue decoding when encountering more errors.Preferably, the faster decoding rate distribution between N−1 number ofdecoders increases exponentially (E.g. a first decoder: real-time rate;second decoder: 2× real-time; third decoder: 4× real-time; fourthdecoder: 8× real-time; fifth decoder: 16× real-time . . . ) up to amaximum rate (e.g. 32× real-time). Preferably, the distribution of themaximum available rate for the N−1 catch up decoders (i.e. the decodersdecoding at a rate faster than the real-time rate) is determined usingthe following formula:

r ¹ +r ² + . . . +r ^((N-1))=maximum available rate

where r is 1/rate as mentioned above. The r value may be used inupdating the expected catch up time in catch up time formulas above inresponse to a change in the number of active decoders. The maximumavailable rate can be dependent on available resources as discussedabove.

When missing packets are received, they are stored in theircorresponding frame container at step 512. At step 513, it is determinedif all of the packets have been received to complete the frame, and ifso, the process moves on to step 514. At step 514, it is determined ifthe second decoder 13 is active, and if not, it is activated at step515. At step 515, the second decoder 13 is initialised with the contextof the first decoder 12 saved at step 511.

Using the packets initially stored at step 503 in the frame containerand the retransmitted/recovered packets stored at step 512 in the sameframe container, the second decoder 13 decodes the complete frame atstep 517. At step 517, the completed frame is decoded at a rate that isfaster than the real-time rate. The second decoder 13 then decodes anysubsequent frames that have been stored at the faster rate until itcatches up with first decoder 12. Once catch up has been completed (step519), the first decoder 12 ceases decoding frames and the second decoder13 decodes frames at the real-time rate, which are then outputted fordisplay instead of the frames from the first decoder 12. The process atthe second decoder 13 then continues for subsequently received framesfrom step 507. If, during the catch up period, the second decoder 13encounters an incomplete frame (step 516), it is determined if anothercontext can be saved (step 510) and a second catch up can begin asdescribed in relation to FIG. 3. If it is determined that adequateresources are not available for another context to be saved, then thecatch up is halted at step 522, the saved context is cleared and an IDRis requested.

At step 520, if the catch up period exceeds a predetermined amount oftime (for example, due to a slowdown in the decode rate due to a lack ofresources), the catch up is halted, the saved context is cleared and anIDR is requested (step 521).

If an IDR frame or I-frame is received at any time (step 507), then anycatch up is halted and any saved contexts are cleared (step 523). If anincomplete I-frame is received, a new context is saved based on theincomplete I-frame. Any subsequent P-frames will be based on theincomplete I-frame and thus will be erroneous as the encoder and decoderare not synchronised. However, this will still allow the P-frames to bedisplayed to maintain the real-time display. Encoder-decodersynchronisation can then be regained using the catch up method describedherein when the missing packets for completing the I-frame are received.The completed I-frame will then be saved as part of new context datawhich is used for catch up at the second decoder 13.

At step 507, a complete loss of an I-frame can also be detected. Framenumbers become reset after each I-frame and then increase for eachsubsequent dependent P-frame. If, at step 507, a P-frame number is lessthan the previously received P-frame then this can indicate that thereshould have been an I-frame received in between those P-frames.Therefore, a loss of an I-frame is detected and an IDR can be requested.

The examples described herein have many advantages over previous methodsfor dealing with errors in video streams. By providing a plurality ofdecoders and a capability to switch the decoded frames to be displayedbetween the plurality of decoders, a stream of video frames for displaycan smoothly transition from incomplete frames being output from onedecoder to complete, error-free frames decoded at another decoder. Thusthe stream of frames need not be halted whilst it is attempted tocomplete the frame, which can cause jerky, slow/fast, non-real-timeplayback and frame freezes. Furthermore, this arrangement reduces theneed for requesting IDRs due to packet losses by relying onre-transmitted packets or error correction data which may arrive laterthan their playback time.

The examples described herein further allow a receiving device todisplay frames at the minimum possible latency, without having to allowfor frames to be completed in the event of an error. A reduction inend-to-end latency typically reduces the buffering requirements, whichcan be important for the user experience during a videoconferencing/telephony. Unlike methods that require both encoder anddecoder support, such as RPS and FEC, the methods described herein needonly be implemented at a decoder (i.e. only at the receiver side).

Data processing systems configured in accordance with the examplesdescribed herein could be embodied in hardware, software or any suitablecombination of hardware and software. A data processing system of theexamples described herein could comprise, for example, software forexecution at one or more processors (such as at a CPU and/or GPU),and/or one or more dedicated processors (such as ASICs), and/or one ormore programmable processors (such as FPGAs) suitably programmed so asto provide functionalities of the data processing system, and/orheterogeneous processors comprising one or more dedicated, programmableand general purpose processing functionalities. In the examplesdescribed herein, data processing systems comprise one or moreprocessors and one or more memories having program code stored thereon,the data processors and the memories being such as to, in combination,provide the claimed data processing systems and/or perform the claimedmethods.

Data processing units described herein (e.g. decoders 12 and 13 andcontroller 14) need not be provided as discrete units and representfunctionalities that could (a) be combined in any manner, and (b)themselves comprise one or more data processing entities. Dataprocessing units could be provided by any suitable hardware or softwarefunctionalities, or combinations of hardware and softwarefunctionalities.

The term software as used herein includes executable code for processors(e.g. CPUs and/or GPUs), firmware, bytecode, programming language codesuch as C or OpenCL, and modules for reconfigurable logic devices suchas FPGAs. Machine-readable code includes software and code for defininghardware, such as register transfer level (RTL) code as might begenerated in Verilog or VHDL.

Any one or more of the data processing methods described herein could beperformed by one or more physical processing units executing programcode that causes the unit(s) to perform the data processing methods.Each physical processing unit could be any suitable processor, such as aCPU or GPU (or a core thereof), or fixed function or programmablehardware. The program code could be stored in non-transitory form at amachine readable medium such as an integrated circuit memory, or opticalor magnetic storage. A machine readable medium might comprise severalmemories, such as on-chip memories, computer working memories, andnon-volatile storage devices.

The applicant hereby discloses in isolation each individual featuredescribed herein and any combination of two or more such features, tothe extent that such features or combinations are capable of beingcarried out based on the present specification as a whole in the lightof the common general knowledge of a person skilled in the art,irrespective of whether such features or combinations of features solveany problems disclosed herein, and without limitation to the scope ofthe claims. The applicant indicates that aspects of the presentinvention may consist of any such individual feature or combination offeatures. In view of the foregoing description it will be evident to aperson skilled in the art that various modifications may be made withinthe scope of the invention.

1. A data processing system for decoding a data stream, the dataprocessing system comprising: a first decoder configured to decode thedata stream into a first sequence of media frames at a first decode rateso as to generate a first media stream for playback in real-time; asecond decoder operable to decode the data stream into a second sequenceof media frames at a second decode rate so as to generate a second mediastream; and a controller configured to detect an error in decoding aparticular frame of the first sequence and, responsive to thatdetection, cause the second decoder to decode the data stream from theparticular frame in the sequence of media frames, the second decoderbeing arranged to decode from the particular frame in dependence onerror correction data received over the data stream for the particularframe, wherein the second decode rate is faster than the first decoderate such that the second sequence of frames of the second media streamcatches up with the first sequence of frames of the first media stream,the controller being further configured to: determine when the secondsequence of frames from the second decoder catches up with the firstsequence of frames from the first decoder; and responsive to thatdetermination: (i) cause the second decoder to operate at the firstdecode rate so as to generate the second media stream for playback inreal-time, and (ii) switch the stream for playback from the firstdecoder to the second decoder.
 2. A data processing system as claimed inclaim 1, the controller being further configured to cause the firstdecoder to cease decoding the media data stream responsive to itsdetermination as to when the sequence of frames from the second decodercatches up with the sequence of frames from the first decoder
 3. A dataprocessing system as claimed in claim 1, wherein the controller isconfigured to determine when the sequence of frames from the seconddecoder catches up with the sequence of frames from the first decoder byestimating from the first and second decode rates when the first andsecond decoders will decode the same frame.
 4. A data processing systemas claimed in claim 1, wherein the controller is configured to determinewhen the sequence of frames from the second decoder catches up with thesequence of frames from the first decoder by determining a timedifference between a time when the particular frame is decoded at thefirst decoder and a time when the particular frame is decoded at thesecond decoder, and dividing the time difference by a relative ratesubtracted by 1, the relative rate being the second decode rate dividedby the first decode rate.
 5. A data processing system as claimed inclaim 1, the controller being configured to determine the second decoderate from the processing resources available at the data processingsystem for the second decoder.
 6. A data processing system as claimed inclaim 1, wherein the controller is configured to, responsive to theerror detection, initialise the second decoder using context data fromthe first decoder.
 7. A data processing system as claimed in claim 6,wherein the context data includes one or more of the last completeframes of the first media stream, state information of the first decoderup to the point at which the last complete frame was decoded, the firstframe as generated by the first decoder, and state information of thefirst decoder from the point at which the error was encountered.
 8. Adata processing system as claimed in claim 1, wherein the errorencountered by the first decoder is data missing from the data stream,or corrupted data of the data stream, or an encoder error.
 9. A dataprocessing system as claimed in claim 1, the second decode rate beingbetween two and four times greater than the first rate.
 10. A dataprocessing system as claimed in claim 1, the second decode rate beingabout two times greater than the first rate.
 11. A data processingsystem as claimed in claim 1, the first decoder being configured todecode said particular frame so as to generate a frame with artefactsfor playback.
 12. A data processing system as claimed in claim 1, thefirst decoder being configured to, on continuing to generate the firstmedia stream for playback after encountering the error decoding theparticular frame, decode frames subsequent to the particular frame so asto generate frames for playback which potentially include artifacts. 13.A data processing system as claimed in claim 1, further comprising acommunications port for receiving the data stream.
 14. A method ofdecoding a data stream, the method comprising: at a first decoder,decoding the data stream into a first sequence of media frames at afirst decode rate so as to generate a first media stream for playback inreal-time; detecting an error in decoding a particular frame of thefirst sequence and, responsive to that detection, causing a seconddecoder to decode the data stream from the particular frame in thesequence of media frames, at the second decoder, decoding the datastream into a second sequence of media frames at a second decode rate soas to generate a second media stream, the second decoder decoding fromthe particular frame in dependence on error correction data receivedover the data stream for the particular frame, wherein the second decoderate is faster than the first decode rate such that the second sequenceof frames of the second media stream catches up with the first sequenceof frames of the first media stream; determining when the secondsequence of frames from the second decoder catches up with the firstsequence of frames from the first decoder; and responsive to thatdetermination: (i) causing the second decoder to operate at the firstdecode rate so as to generate the second media stream for playback inreal-time; and (ii) switching the stream for playback from the firstdecoder to the second decoder.
 15. A non-transitory computer readablestorage medium having encoded thereon computer-executable code that whenexecuted causes at least one processor to: decode a data stream into afirst sequence of media frames at a first decode rate so as to generatea first media stream for playback in real-time; decode the data streaminto a second sequence of media frames at a second decode rate so as togenerate a second media stream; detect an error in decoding a particularframe of the first sequence and, responsive to that detection, causedecoding of the data stream from the particular frame into said secondsequence in dependence on error correction data received over the datastream for the particular frame, wherein the second sequence decode rateis faster than the first sequence decode rate such that the secondsequence of frames of the second media stream catches up with the firstsequence of frames of the first media stream, determine when the secondsequence of decoded media frames catches up with the first sequence ofdecoded media frames; and responsive to that determination, cause thesecond media stream to be used for playback in real-time.